Model selection and tuning
See:
Resources
Bias-variance trade-off
- Problem of minimizing two sources of errors that prevent a supervised learning algorithm from generalizing beyond the training set:
- High bias -> underfitting
- High variance -> overfitting
- Validation curves: plotting scores to evaluate models
- https://www.quora.com/How-would-you-explain-the-bias-variance-tradeoff-to-a-five-year-old
- http://scott.fortmann-roe.com/docs/BiasVariance.html
- http://scott.fortmann-roe.com/docs/MeasuringError.html
- https://elitedatascience.com/bias-variance-tradeoff
- Overfitting
Cross-validation
- Train, test and validation
- Train, test, validation split and cross-validation (scikit-learn documentation)
- http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc_crossval.html
- https://sebastianraschka.com/blog/2016/model-evaluation-selection-part3.html
- https://blog.dataiku.com/model-sucks-evaluating-models-validation-set-infographic
- Making Predictive Models Robust: Holdout vs Cross-Validation
- How to Train a Final ML Model
- http://nbviewer.jupyter.org/github/cs109/content/blob/master/lec_10_cross_val.ipynb
- Hyperparameter optimization
- Tuning the hyper-parameters of an estimator (scikit-learn documentation)
Code
- See AI/Data Engineering/ML Ops
- #CODE Vizier - Python-based research interface for blackbox and hyperparameter optimization, based on the internal Google Vizier Service
- https://ai.googleblog.com/2023/02/open-source-vizier-towards-reliable-and.html
- #PAPER Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimization (Song 2023)
- https://www.marktechpost.com/2023/02/04/google-ai-open-sources-vizier-a-standalone-python-package-designed-for-managing-and-optimizing-machine-learning-experiments-at-scale/
- #CODE Optuna - A hyperparameter optimization framework
- #CODE Yellowbrick - Visual analysis and diagnostic tools to facilitate machine learning model selection
- #CODE Tune-sklearn - Drop-in replacement for Scikit-Learn’s model selection module (GridSearchCV, RandomizedSearchCV) with cutting edge hyperparameter tuning techniques
- #CODE Talos - Hyperparameter Optimization for TensorFlow, Keras and PyTorch
- #CODE Hyperopt - Distributed Asynchronous Hyperparameter Optimization in Python
- #CODE Hyperband - A Novel Bandit-Based Approach to Hyperparameter Optimization